If the tokenizer used for word tokenization knows abbreviations (ending with full stop; only those abbreviations may yield to problems during sentence segmentation), these are included in the word list as words ending in a full stop. The list shows the 50 most frequent words of this form.
The list should contain the most frequent abbreviations ending with full stop. If important abbreviations are missing, they might be considered as sentence boundaries by the sentence segmentizer.
If the list is generated automatically, it also may contain non-abbreviations. There may be missing sentence boundaries at the corresponding positions.
The confusion between abbreviations and sentence boundaries exists only for those languages which use the same punctuation mark for both. This is not the case for languages like Chinese or Japanese.
select w_id-100,word,freq from words where word like "%." and word not regexp "^[0-9]*\.$" limit 30;